Pilot-Streaming: Design Considerations for a Stream Processing Framework for High-Performance Computing

Streaming capabilities are becoming increasingly important for scientific applications [1], [2] supporting important needs, such as the ability to act on incoming data and steering. The interoperable use of streaming data sources within HPC environments is a critical capability for an emerging set of applications. Scientific instruments, such as x-ray light sources (e. g., the Advanced Photon Source (APS) and the Advanced Light Source (ALS) [3]), can generate large amounts of highvelocity data in a diverse set of experiments. Coupling data streams produced by such experiments to computational HPC capabilities is an important challenge. Supporting the processing of high data rates streams executing, e. g., predictions and outlier detection algorithms on it, while running larger models in batch mode on the entire dataset, is a challenging task. The increasing demands lead to a heterogeneous landscape of infrastructures and tools supporting streaming needs on different levels. Batch frameworks, such as Spark [4] have been extended to provide streaming capabilities [5], while different native streaming frameworks, such as Storm [6] and Flink [7] have emerged. We define a streaming application as an application that processes and acts on real-time data, also referred to as event stream. Different usage modes for stream processing can be observed: • Coordination: Usage of stream processing to connect a data source and data analysis phase. Sometime this includes the pre-processing and transformation of the data before it becomes persistent (e. g. the Hadoop Filesystem (HDFS)) and analyzed (e. g. using a Hadoop processing framework). • Realtime Analytics: This type of application utilizes machine learning on incoming data, e. g. for scoring, classification or outlier detection. • Analytics and Model Update: The applications combine stream processing with other forms of processing, e. g. the continuous update of a machine learning model on historical data and real-time scoring/classification or a simulation. The complexity of the application increases from the top to bottom. In the last application type streaming, batch and interactive processes utilizing different abstractions and runtime systems need to be combined, algorithms need to be adapted to process windows of data instead of the complete bounded data-set. There are several challenges that need to be address when developing streaming applications: • Infrastructure: How to efficiently handle data streams (delivery guarantees, low latencies, varying data rate)? How to decouple data producer and consumer? How to store data to allow flexible stream and batch processing (event stream as a log, random access and mutable storage)? • Abstractions: How to decouple application concerns from streaming infrastructures? How can high-level abstraction, such as SQL oder data pipelines be efficiently supported in streaming mode? • Applications: The application itself needs to be capable of utilizing streaming data. Often, the algorithm needs to be adapted to meaningful incorporate incoming data. While batch algorithms assume that they operate on a complete dataset, streaming applications need to operate on a window of data (e. g. a fixed, sliding or session window) [8]. Thus, it is often necessary to balance historical and recent data in machine learning algorithms using e.g. decay factors. Simulations e. g. need an algorithm for including streaming data into their current state.